Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

157-160: Document rationale for deterministic two-step install approach.

Lines 157–160 implement a specific install sequence for causal-conv1d and mamba-ssm with --no-build-isolation and --no-deps flags. While this pattern may align with runtime behavior, add a comment explaining:

• Why causal-conv1d must be installed first
• Why mamba-ssm requires --no-deps
• Whether this order prevents dependency conflicts or resolves specific build issues

~# Deterministic 2-step: sub-dep first, then parent without deps (align with runtime)
+# Deterministic 2-step: sub-dep first, then parent without deps (align with runtime)
+# causal-conv1d must be installed before mamba-ssm to ensure it is available during mamba-ssm build.
+# mamba-ssm uses --no-deps to prevent pip from installing a conflicting version of causal-conv1d.
 RUN pip install --retries 5 --timeout 300 --no-cache-dir --no-build-isolation causal-conv1d==1.5.3.post1 && \
     pip install --retries 5 --timeout 300 --no-cache-dir --no-build-isolation mamba-ssm==2.2.6.post3 --no-deps && \
     fix-permissions /opt/app-root -P

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

137-137: Upgrade bitsandbytes to 0.48.2 — version 0.48.1 lacks CUDA 12.8 wheels

bitsandbytes 0.48.1 has no CUDA 12.8 wheels available on PyPI (0 cu128 wheels found). This will cause runtime failures when the container attempts to load libbitsandbytes_cuda128. Upgrade to bitsandbytes 0.48.2, which has confirmed cu128 support.

-    bitsandbytes==0.48.1 \
+    bitsandbytes==0.48.2 \

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

73-76: Avoid referencing environment variables within the same ENV statement (Hadolint DL3044).

While Docker expands variables in this context, it is clearer and follows Dockerfile best practices to split the definition into separate statements.

Apply this diff to separate the ENV statements:

-ENV CUDA_HOME=/usr/local/cuda \
-    PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-    TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ENV CUDA_HOME=/usr/local/cuda
+
+ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
+    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH} \
+    TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

67-70: Missing ldconfig call—unresolved critical issue from prior review.

Lines 67–68 append CUDA library paths to /etc/ld.so.conf.d/nvidia.conf, but the dynamic linker cache is not rebuilt. The ldconfig -p check on line 82 comes too late and does not rebuild the cache. Without an explicit ldconfig call immediately after writing the config, CUDA libraries at /usr/local/nvidia/lib and /usr/local/nvidia/lib64 remain undiscoverable during preflight checks and at runtime.

Apply this diff to rebuild the cache immediately:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

142-142: Upgrade bitsandbytes to 0.48.2 or later—unresolved critical issue from prior review.

Line 142 pins bitsandbytes to 0.48.1, which lacks official CUDA 12.8 wheels. Versions 0.48.2 and later include published wheels with proper CUDA 12.8 support. Without this upgrade, the image will fail at runtime when bitsandbytes attempts to load libbitsandbytes_cuda128.

Apply this diff:

-    bitsandbytes==0.48.1 \
+    bitsandbytes==0.48.2 \

---

183-193: Remove --no-deps to enable dependency validation—unresolved critical issue from prior review.

Line 189 uses --no-deps when building mamba-ssm, which bypasses pip's dependency resolution and prevents verification that causal-conv1d and mamba-ssm versions are compatible. While the 2-step deterministic build with verbose logging is good for debugging, using --no-deps masks potential missing or incompatible dependencies.

Apply this diff to allow pip to validate the dependency chain:

- pip wheel --no-cache-dir --no-build-isolation -vv \
-   --log /tmp/pip-mamba-ssm.log \
-   mamba-ssm==2.2.6.post3 --no-deps -w /tmp/wheels || \
+ pip wheel --no-cache-dir --no-build-isolation -vv \
+   --log /tmp/pip-mamba-ssm.log \
+   causal-conv1d==1.5.3.post1 mamba-ssm==2.2.6.post3 -w /tmp/wheels || \

Then update line 191 accordingly:

- pip install --no-cache-dir /tmp/wheels/*.whl && \
+ pip install --no-cache-dir /tmp/wheels/mamba*.whl && \

Alternatively, add causal-conv1d==1.5.3.post1 to your training-hub requirements so pip resolves both transitively in a single install step.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (4)

67-70: ⚠️ [Unresolved] Missing ldconfig call after appending to /etc/ld.so.conf.d/nvidia.conf.

Writing to ldconfig config files does not take effect until ldconfig is run to rebuild the cache. The current code appends to /etc/ld.so.conf.d/nvidia.conf but does not call ldconfig, so the CUDA libraries at /usr/local/nvidia/lib and /usr/local/nvidia/lib64 will not be discoverable during the runtime checks on line 82 or at runtime. This was flagged in a prior review and remains unresolved.

Apply this diff to add the missing ldconfig call:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

73-76: ⚠️ [Unresolved] ENV statement violates Hadolint DL3044 with self-referencing variables.

Lines 74–75 reference ${PATH} and ${LD_LIBRARY_PATH} within the same ENV statement where they are being defined. This violates Docker best practices (Hadolint DL3044) and may cause unexpected behavior. This issue was flagged in a prior review and remains unresolved.

Refactor into separate ENV statements to allow prepending to existing values:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

---

150-150: ⚠️ [Unresolved] Upgrade bitsandbytes from 0.48.1 to 0.48.2 or later for CUDA 12.8 wheel support.

Line 150 pins bitsandbytes==0.48.1, which lacks official CUDA 12.8 wheels. Bitsandbytes 0.48.2 and later include published wheels targeting CUDA 12.8, resolving runtime failures when loading libbitsandbytes_cuda128. This critical issue was flagged in a prior review and remains unresolved.

Apply this diff to upgrade the package:

- bitsandbytes==0.48.1 \
+ bitsandbytes==0.48.2 \

---

191-201: ⚠️ [Unresolved] mamba-ssm build still uses --no-deps, bypassing dependency validation.

Line 197 continues to use --no-deps when building the mamba-ssm wheel, which bypasses pip's dependency resolution and prevents verification that optional dependencies (like causal-conv1d) are compatible with the installed version. This critical issue was flagged in a prior review and remains unresolved.

While the 2-step deterministic build pattern (build wheel, then install from local wheel) is a reasonable approach for reproducibility and debugging, the --no-deps flag introduces risk.

Recommended fix: Install both packages together in a single pip invocation to let pip resolve compatibility:

- RUN python -m pip install --no-cache-dir --no-build-isolation -vv --log /tmp/pip-causal.log causal-conv1d==1.5.3.post1 && \
-     CMAKE_GENERATOR=Ninja CMAKE_ARGS="-DCMAKE_VERBOSE_MAKEFILE=ON" \
-     python -m pip wheel --no-cache-dir --no-build-isolation -vv \
-       --log /tmp/pip-mamba-ssm.log \
-       mamba-ssm==2.2.6.post3 --no-deps -w /tmp/wheels || \
-     (echo '--- pip mamba-ssm log (tail) ---'; tail -n 500 /tmp/pip-mamba-ssm.log; exit 1) && \
-     python -m pip install --no-cache-dir /tmp/wheels/*.whl && \
-     rm -rf /tmp/wheels && \
-     fix-permissions /opt/app-root -P
+ RUN CMAKE_GENERATOR=Ninja CMAKE_ARGS="-DCMAKE_VERBOSE_MAKEFILE=ON" \
+     python -m pip install --retries 5 --timeout 300 --no-cache-dir --no-build-isolation \
+       causal-conv1d==1.5.3.post1 mamba-ssm==2.2.6.post3 && \
+     fix-permissions /opt/app-root -P

Alternatively, if deterministic wheel builds are essential, remove --no-deps to allow pip to validate both packages together before installation.

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

108-110: Consider removing or conditionally gating diagnostic print statements.

Lines 108–110 print Python interpreter details and torch metadata during the build. These lines are useful for build-time debugging but are unusual in a production Dockerfile. Consider either removing them, moving them into a separate build stage, or gating them behind a build ARG for optional diagnostic mode.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

150-150: Resolve unaddressed critical issue: upgrade bitsandbytes to 0.48.2+ for CUDA 12.8 support.

Line 150 pins bitsandbytes to 0.48.1, which was flagged as a critical issue in prior reviews and remains unresolved. Version 0.48.1 lacks official CUDA 12.8 wheels; the runtime will fail when trying to load libbitsandbytes_cuda128. A prior review confirmed via web search that bitsandbytes 0.48.2 and later include published wheels targeting CUDA 12.8.

Apply this fix:

-    bitsandbytes==0.48.1 \
+    bitsandbytes==0.48.2 \

---

67-70: Add ldconfig call to rebuild dynamic linker cache after updating ld.so.conf.d.

Lines 67–68 append NVIDIA library paths to /etc/ld.so.conf.d/nvidia.conf, but the dynamic linker cache is not rebuilt. This was flagged as a major issue in prior reviews. Without calling ldconfig, the CUDA libraries at /usr/local/nvidia/lib and /usr/local/nvidia/lib64 remain undiscoverable, and the preflight check at line 82 may fail.

Apply this diff:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

73-76: Split ENV statement to avoid self-referencing variables (Hadolint DL3044).

Lines 73–76 define CUDA_HOME, PATH, LD_LIBRARY_PATH, and TORCH_CUDA_ARCH_LIST in a single ENV block, with PATH and LD_LIBRARY_PATH self-referencing their own values. This violates Hadolint DL3044 and may behave unexpectedly. This was flagged as a major issue in prior reviews and remains unresolved.

Refactor into separate ENV statements to allow safe prepending:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}
+ ENV TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

189-199: Consider removing or better-documenting the large commented-out code block.

Lines 189–199 contain a commented-out 2-step deterministic build for causal-conv1d and mamba-ssm with verbose logging. If these packages are not planned for immediate use, remove this block to keep the Dockerfile clean. If these are deferred features, add a clearer comment explaining the reason for deferral (e.g., "# TODO: Enable after resolving mamba-ssm --no-deps issue" or "# DEBUG: Disabled pending package availability").

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (4)

177-177: Critical: Update bitsandbytes to version 0.48.2 or later (duplicate fix required).

This is the second occurrence of bitsandbytes==0.48.1 that must be updated to match the fail-fast check. See the preceding comment for full context.

Apply this diff:

- bitsandbytes==0.48.1 \
+ bitsandbytes==0.48.2 \

---

151-152: Critical: Update bitsandbytes to version 0.48.2 or later for CUDA 12.8 support.

bitsandbytes 0.48.1 publishes only generic manylinux wheels without CUDA 12.8-specific wheels on PyPI. Since this Dockerfile targets CUDA 12.8, the fail-fast check here will fail during installation, and even if it passes, runtime will fail when attempting to load libbitsandbytes_cuda128. This issue was previously flagged as critical and marked as resolved, but the version remains unchanged.

Apply this diff:

- python -m pip download --retries 5 --timeout 120 --only-binary=:all: --no-deps -d /tmp/wheels \
-   numba==0.62.1 bitsandbytes==0.48.1 || \
+ python -m pip download --retries 5 --timeout 120 --only-binary=:all: --no-deps -d /tmp/wheels \
+   numba==0.62.1 bitsandbytes==0.48.2 || \

---

68-71: Add ldconfig call after updating /etc/ld.so.conf.d/nvidia.conf.

Writing to ldconfig config files does not take effect until ldconfig is run to rebuild the cache. Without this step, the CUDA libraries at /usr/local/nvidia/lib and /usr/local/nvidia/lib64 will not be discoverable during the preflight checks on line 88 or at runtime.

Apply this diff to add the ldconfig call:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

73-76: Fix environment variable self-reference (Hadolint DL3044).

Within a single ENV statement, you cannot reference a variable being defined (e.g., ${PATH} and ${LD_LIBRARY_PATH} on lines 75-76). This violates Docker best practices. Refactor into separate ENV statements to allow prepending to existing values.

Apply this diff to split the ENV block:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH 
-     # TODO: revisit this, going with default list for now.
-     # TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

Also uncomment TORCH_CUDA_ARCH_LIST (currently commented on line 78) to enable explicit CUDA architecture targeting for builds.

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

228-264: Clarify rationale for mamba-ssm --no-deps pattern.

The 2-step installation of causal-conv1d followed by mamba-ssm with --no-deps (line 241) bypasses pip's dependency resolution. While this can improve determinism and control build order, it prevents pip from verifying compatibility with optional components. If causal-conv1d is missing, mamba-ssm falls back to a slower eager implementation rather than failing fast.

Consider documenting in a comment why this explicit 2-step approach is necessary (e.g., "avoids pip resolver conflicts with training-hub" or similar), or alternatively, declare causal-conv1d==1.5.3.post1 explicitly in the training-hub requirements so pip can resolve both transitively with proper version constraints.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (2)

68-71: Add ldconfig call to make /etc/ld.so.conf.d/nvidia.conf paths discoverable.

Add a file with the .conf extension to /etc/ld.so.conf.d/ that contains the paths to the libraries and then run ldconfig. Modifying /etc/ld.so.conf.d/nvidia.conf alone does not take effect until ldconfig is invoked to rebuild the dynamic linker cache. The preflight check at line 88 (ldconfig -p | grep) will fail if ldconfig is not run after the echo commands.

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

74-76: Split ENV statements to resolve self-reference violation (Hadolint DL3044).

Docker will not expand a variable within the same ENV statement where it is defined. While it will not crash Docker, it carries a high likelihood of errors. Lines 74-76 reference ${PATH} and ${LD_LIBRARY_PATH} in the same ENV statement where they are being prepended, violating Hadolint's DL3044 rule.

Refactor into separate ENV statements:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH 
-     # TODO: revisit this, going with default list for now.
-     # TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

Also note: the TODO comment about TORCH_CUDA_ARCH_LIST was removed; ensure this is intentional and the default Torch compile list is acceptable for your GPUs.

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

228-282: Simplify or document the mamba-ssm wheel build logic.

Lines 228-282 implement a complex heredoc-based shell script embedded in a single RUN instruction to build the mamba-ssm wheel with timeout and heartbeat monitoring. While the intent (prevent hangs, log progress) is sound, the 50+ lines of inline shell logic make the Dockerfile harder to maintain. Consider:

1. Extract to a separate build script in the repository (e.g., build-mamba-ssm.sh) and COPY it into the image before running it. This improves readability and allows unit testing of the script.
2. Add a brief comment explaining why the 2-step causal-conv1d + mamba-ssm approach is necessary (e.g., "Mamba-SSM build takes >10 minutes; timeout ensures the process completes or fails safely without hanging the build").
3. Verify the --no-deps flag on line 280 is intentional—are there any transitive dependencies of mamba-ssm beyond causal-conv1d that should be installed?

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

151-151: Update bitsandbytes to 0.48.2 or later (critical, unresolved from prior reviews).

Line 151 still pins bitsandbytes==0.48.1, which lacks official CUDA 12.8 wheels on PyPI. This was flagged as critical in prior reviews and will cause runtime failures when attempting to load libbitsandbytes_cuda128. Upgrade to 0.48.2 or later for cu128 support:

- numba==0.62.1 bitsandbytes==0.48.1 || \
+ numba==0.62.1 bitsandbytes==0.48.2 || \

Also update line 177:

- bitsandbytes==0.48.1 \
+ bitsandbytes==0.48.2 \

---

68-71: Missing ldconfig call after updating /etc/ld.so.conf.d/nvidia.conf.

Writing to ldconfig config files requires rebuilding the dynamic linker cache for the paths to become discoverable. The preflight check on line 88 (ldconfig -p | grep -E) will fail without this step.

Apply this diff to add the ldconfig call:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

74-76: Fix Hadolint DL3044: split self-referencing ENV variables into separate statements.

Referencing ${PATH} and ${LD_LIBRARY_PATH} within the same ENV block where they are being defined violates Docker best practices. Split these into separate ENV statements:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH 
-     # TODO: revisit this, going with default list for now.
-     # TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

228-285: Consider removing --no-deps from mamba-ssm and installing both packages together.

Line 243 uses --no-deps to skip dependency validation, relying on causal-conv1d being pre-installed on line 228. While this works because causal-conv1d is indeed installed first, it bypasses pip's dependency resolver, preventing verification of version compatibility.

The complex shell wrapper adds valuable diagnostics and timeout handling, but the underlying pattern could be improved. Consider this alternative:

- RUN python -m pip install --no-cache-dir --no-build-isolation -vv --log /tmp/pip-causal.log causal-conv1d==1.5.3.post1 && \
+ RUN python -m pip install --no-cache-dir --no-build-isolation causal-conv1d==1.5.3.post1 && \
    [shell script with diagnostics/timeout] && \
-     python -m pip install --no-cache-dir /tmp/wheels/*.whl && \
+     python -m pip install --no-cache-dir /tmp/wheels/*.whl --no-deps && \

Or better yet: add causal-conv1d==1.5.3.post1 to your training-hub requirements so pip resolves both transitively without --no-deps.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

68-71: Missing ldconfig call after updating /etc/ld.so.conf.d/nvidia.conf.

Writing to ldconfig config files does not take effect until ldconfig is run to rebuild the cache. Lines 68–69 append NVIDIA library paths to the config, but the cache is never rebuilt. Without this step, the CUDA libraries at /usr/local/nvidia/lib and /usr/local/nvidia/lib64 will not be discoverable at runtime or during the preflight check on line 88.

Apply this diff to rebuild the ldconfig cache:

 && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
 && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
 && dnf clean all \

---

183-183: Upgrade bitsandbytes to 0.48.2 or later for CUDA 12.8 compatibility.

Line 183 pins bitsandbytes==0.48.1, which was flagged as incompatible with CUDA 12.8 in prior reviews. Ensure consistency: if the fail-fast check on line 157 is updated to a newer version, this line must also be updated.

Apply this diff:

-    bitsandbytes==0.48.1 \
+    bitsandbytes==0.48.2 \

---

75-78: Fix Hadolint DL3044: ENV statement cannot self-reference its own variables.

Lines 75–78 violate Docker best practices (and trigger Hadolint DL3044) by referencing ${PATH} and ${LD_LIBRARY_PATH} within the same ENV statement where they are being set. This can cause unexpected behavior depending on the Docker version.

Refactor into separate ENV statements to allow prepending to existing values:

-ENV CUDA_HOME=/usr/local/cuda \
-    PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-    TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ENV CUDA_HOME=/usr/local/cuda \
+    TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

images/universal/training/py312-cuda128-torch280/Dockerfile (2)

163-163: Simplify the find command to avoid Shellcheck SC2261 warning.

Hadolint and Shellcheck flag line 163 for multiple redirections competing for stdout. This occurs within the fail-fast check when running find /tmp -type f -name "*.o" | wc -l || true. While the || true fallback handles errors gracefully, a cleaner pattern would silence the output on failure:

-      numba==0.62.1 bitsandbytes==0.48.1 || \
+      numba==0.62.1 bitsandbytes==0.48.2 || \

(Note: Once bitsandbytes is upgraded to 0.48.2 per the critical issue above, this refactor becomes moot.)

---

232-316: Mamba-SSM wheel build section: extensive logging is welcome, but document the timeout logic.

Lines 232–316 implement an impressive deterministic 2-step wheel build with ccache support, detailed heartbeat logging, and a 20-minute timeout to prevent runaway compilation. The timeout and process-group cleanup logic are robust. However, the inline shell script (generated dynamically) is difficult to debug and maintain.

Recommendations for a future refactor:

• Extract the mamba-ssm build script into a separate file in the Dockerfile directory (e.g., build-mamba-ssm.sh) and COPY it into the image, making it easier to version control and debug.
• Add a comment explaining why the 20-minute timeout is chosen (e.g., "Historical data shows mamba-ssm 2.2.6.post3 compiles in ~8–15 minutes on c5.4xlarge; 20 minutes provides 33% headroom").
• Document why --no-deps is used for mamba-ssm and whether causal-conv1d is a hard requirement or optional fallback.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (2)

68-71: Add ldconfig call to rebuild dynamic linker cache after updating /etc/ld.so.conf.d/.

Lines 68–71 append NVIDIA library paths to /etc/ld.so.conf.d/nvidia.conf, but the dynamic linker cache is not rebuilt. Without ldconfig, CUDA libraries will not be discoverable at line 88 (ldconfig grep verification) or at runtime. This was flagged in prior reviews.

Apply this diff:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

157-157: Critical regression: bitsandbytes 0.48.1 lacks CUDA 12.8 wheels—upgrade required.

Lines 157 and 183 still pin bitsandbytes==0.48.1, which lacks CUDA 12.8 (cu128) wheel support. The fail-fast check at line 157 will fail during image build, and at runtime, the library fails to load libbitsandbytes_cuda128.

Upgrade to bitsandbytes==0.48.2 or later (verified cu128 support):

Line 157:

- numba==0.62.1 bitsandbytes==0.48.1 || \
+ numba==0.62.1 bitsandbytes==0.48.2 || \

Line 183:

- bitsandbytes==0.48.1 \
+ bitsandbytes==0.48.2 \

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

232-235: Clarify the 2-step causal-conv1d + mamba-ssm --no-deps pattern.

Lines 232–235 use a 2-step installation with --no-deps on mamba-ssm, which bypasses pip's dependency verification. While this pattern can improve build determinism, document why this explicit approach is necessary (e.g., "causal-conv1d is optional for mamba-ssm and required for performance; using --no-deps avoids resolver conflicts with training-hub").

Alternatively, declare causal-conv1d==1.5.3.post1 in project requirements so pip resolves both transitive versions with proper constraints.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

68-71: Add ldconfig call after updating /etc/ld.so.conf.d/nvidia.conf.

Lines 68–69 append NVIDIA library paths to ldconfig config, but the dynamic linker cache is not rebuilt. Without ldconfig, the CUDA libraries will not be discoverable at line 88 (ldconfig grep check) or at runtime. This was flagged in prior reviews but remains unresolved.

Apply this diff:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

75-78: Hadolint DL3044 violation: ENV statement self-references variables being defined.

Lines 75–78 violate Docker best practices by referencing ${CUDA_HOME} and ${LD_LIBRARY_PATH} within the same ENV statement where they are being defined. This was flagged in prior reviews but remains unresolved.

Split into separate ENV statements:

+ ARG CUDA_ARCH_LIST=9.0
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}
- ARG CUDA_ARCH_LIST=9.0
- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"

---

157-157: Update bitsandbytes from 0.48.1 to 0.48.2 at lines 157 and 183.

bitsandbytes 0.48.1 has a confirmed packaging/build problem with CUDA 12.8 wheels, and 0.48.2 contains the fix. The Dockerfile must apply the suggested diffs to both lines:

Line 157:

- numba==0.62.1 bitsandbytes==0.48.1 || \
+ numba==0.62.1 bitsandbytes==0.48.2 || \

Line 183:

- bitsandbytes==0.48.1 \
+ bitsandbytes==0.48.2 \

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (2)

68-71: Add ldconfig call after updating /etc/ld.so.conf.d/nvidia.conf.

Lines 68–69 append NVIDIA library paths to ldconfig config, but the dynamic linker cache is not rebuilt. Without ldconfig, the CUDA libraries will not be discoverable at line 88 (ldconfig grep check) or at runtime. This was flagged in prior reviews but remains unresolved.

Apply this diff:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

75-78: Hadolint DL3044 violation: ENV statement self-references variables being defined.

Lines 75–78 violate Docker best practices by referencing ${CUDA_HOME} and ${LD_LIBRARY_PATH} within the same ENV statement where they are being defined. This was flagged in prior reviews but remains unresolved.

Split into separate ENV statements:

+ ARG CUDA_ARCH_LIST=9.0
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}
- ARG CUDA_ARCH_LIST=9.0
- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"

images/universal/training/py312-cuda128-torch280/Dockerfile (1)

232-235: Verify mamba-ssm install pattern and document rationale.

The two-step install (causal-conv1d then mamba-ssm --no-deps) assumes causal-conv1d satisfies all of mamba-ssm's dependencies. While this pattern can improve determinism, using --no-deps bypasses pip's dependency verification, which carries risk if mamba-ssm has undeclared dependencies or if version incompatibilities arise.

Consider adding a comment explaining why this explicit pattern is necessary (e.g., "avoids resolver conflicts with training-hub") or refactor to a single install with both packages declared.

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (1)

232-316: Complex mamba-ssm build wrapper in debug variant — verify necessity and maintainability.

This file's lines 232–316 introduce a 84-line shell script that wraps the mamba-ssm wheel build with heartbeat logging, timeout enforcement (20 min), and process monitoring. This is significantly more elaborate than the simple 3-line install pattern in the main Dockerfile (lines 232–235).

Questions:

1. Is the heartbeat/timeout wrapper necessary only for the debug variant, or should it also be in the production Dockerfile?
2. Does the script properly handle edge cases (e.g., timeouts during ccache setup)?
3. Is the 20-minute hardcoded timeout appropriate for all platforms/architectures?

Recommendation: Add a comment explaining the rationale (e.g., "Debug builds may run longer due to extended diagnostics; timeout prevents hung builds") and consider whether similar protections are needed in the production build.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (2)

68-71: ⚠️ CRITICAL: Missing ldconfig call after updating ld.so.conf.d.

Lines 68–71 append NVIDIA library paths to /etc/ld.so.conf.d/nvidia.conf, but do not rebuild the dynamic linker cache. Without running ldconfig, the CUDA libraries will not be discoverable at the preflight check (line 88) or at runtime. The fail-fast check will fail immediately.

Apply this diff to add ldconfig:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

75-78: ⚠️ CRITICAL: Resolve Hadolint DL3044 — ENV self-references variables being defined.

Lines 75–78 violate Docker best practice by referencing ${CUDA_HOME} within the same ENV statement where it is being defined. Additionally, ${LD_LIBRARY_PATH} on line 77 references itself within the same statement. This prevents Docker from properly expanding these values and is flagged by Hadolint DL3044.

Refactor into separate ENV statements:

- ARG CUDA_ARCH_LIST=9.0
- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ARG CUDA_ARCH_LIST=9.0
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

images/universal/training/py312-cuda128-torch280/Dockerfile (2)

68-71: ⚠️ CRITICAL: Missing ldconfig call after updating ld.so.conf.d.

Lines 68–71 append NVIDIA library paths to /etc/ld.so.conf.d/nvidia.conf, but do not rebuild the dynamic linker cache. Without running ldconfig, the CUDA libraries will not be discoverable at the preflight check (line 88) or at runtime. The fail-fast check will fail immediately.

Note: This issue was flagged in prior reviews as addressed (commit a8ad8e6) but remains unresolved in the current code.

Apply this diff to add ldconfig:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

75-78: ⚠️ CRITICAL: Resolve Hadolint DL3044 — ENV self-references variables being defined.

Lines 75–78 violate Docker best practice by referencing ${CUDA_HOME} within the same ENV statement where it is being defined. Additionally, ${LD_LIBRARY_PATH} on line 77 references itself within the same statement. This prevents Docker from properly expanding these values and is flagged by Hadolint DL3044.

Note: This issue was flagged in prior reviews as addressed (commit a8ad8e6) but remains unresolved in the current code.

Refactor into separate ENV statements:

- ARG CUDA_ARCH_LIST=9.0
- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ARG CUDA_ARCH_LIST=9.0
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile (3)

69-72: ** Split ENV statement to avoid self-referencing variables (Hadolint DL3044).**

This issue was flagged in prior reviews but remains unresolved. The single ENV statement self-references ${PATH} and ${LD_LIBRARY_PATH}, which violates Dockerfile best practices. Split into separate statements:

- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/cuda/bin:$PATH \
-     LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"
+ ENV CUDA_HOME=/usr/local/cuda
+ ENV PATH=/usr/local/cuda/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}
+ ENV TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"

---

140-140: ** Upgrade bitsandbytes to 0.48.2 or later (critical CUDA 12.8 support).**

bitsandbytes 0.48.1 lacks official CUDA 12.8 wheels on PyPI and will fail at runtime. This critical issue was flagged in multiple prior reviews and commits. Upgrade to 0.48.2 or later, which provides confirmed cu128 support.

-    bitsandbytes==0.48.1 \
+    bitsandbytes==0.48.2 \

Also update line 155 (pip download step) if present.

---

167-170: ** Remove --no-deps flag and install both packages in single command.**

The --no-deps flag on line 169 bypasses pip's dependency resolution, preventing verification that causal-conv1d and mamba-ssm are compatible. This pattern was flagged in prior reviews. Install both together without --no-deps to enable proper validation:

- # Deterministic 2-step: sub-dep first, then parent without deps
- RUN pip install --no-build-isolation --no-cache-dir causal-conv1d==1.5.3.post1 && \
-     pip install --no-build-isolation --no-cache-dir mamba-ssm==2.2.6.post3 --no-deps && \
+ # Install mamba-ssm with its optional causal-conv1d dependency
+ RUN pip install --no-build-isolation --no-cache-dir causal-conv1d==1.5.3.post1 mamba-ssm==2.2.6.post3 && \
      fix-permissions /opt/app-root -P

images/universal/training/py312-cuda128-torch280/Dockerfile (2)

22-23: Resolve the LICENSE.md TODO before release.

The commented-out COPY instruction should either be re-enabled with the file added to the repo, or removed entirely if licensing is handled elsewhere. Clarify the intent and complete the task.

Do you want me to help you either generate a suitable LICENSE.md file or remove this TODO entirely?

---

161-166: Remove duplicate TORCH_CUDA_ARCH_LIST and pin versions for build helpers.

Line 165 duplicates the TORCH_CUDA_ARCH_LIST already defined on line 72. Additionally, line 162 uses pip install -U which unpins versions and can cause drift; provide explicit versions for reproducibility.

  # Build helpers to compile PyTorch extensions
- RUN pip install -U pip setuptools wheel ninja cmake
+ RUN pip install --no-cache-dir pip setuptools wheel ninja cmake
  
- # Optional: set GPU archs if you hit arch issues
- ENV TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0"

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (1)

75-78: Resolve Hadolint DL3044: Split self-referencing ENV statement.

Lines 75–78 violate Docker best practices (Hadolint DL3044) by referencing ${CUDA_HOME}, ${PATH}, and ${LD_LIBRARY_PATH} within the same ENV statement where they are being defined. This prevents proper variable expansion and is unreliable across Docker versions.

Refactor into separate ENV statements:

- ARG CUDA_ARCH_LIST=9.0
- ENV CUDA_HOME=/usr/local/cuda \
-     PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-     LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ARG CUDA_ARCH_LIST=9.0
+ ENV CUDA_HOME=/usr/local/cuda \
+     TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (1)

234-316: Consider extracting complex wheel-build logic into a separate shell script for maintainability.

Lines 234–316 contain a highly complex multi-line RUN statement that uses printf to generate an inline shell script with ccache logic, timeouts, heartbeat monitoring, and error handling. While the defensive timeout logic and diagnostics are valuable, embedding this complexity makes the Dockerfile harder to audit and maintain.

Suggested improvement (for a follow-up refactor): Extract the script generation and execution into a separate build script file (e.g., build-mamba-wheel.sh), then invoke it from the Dockerfile. This would:

• Separate build logic from container configuration
• Improve readability and testability
• Enable syntax highlighting and linting on the script separately
• Simplify the Dockerfile layer

For now, the current approach is functional. No blocking issue, but worth considering for future maintenance.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (3)

68-71: Add ldconfig call to rebuild dynamic linker cache after updating /etc/ld.so.conf.d/nvidia.conf. Lines 68–71 append NVIDIA library paths to the ldconfig configuration file, but the dynamic linker cache is not rebuilt. Without running ldconfig, the CUDA libraries at those paths will not be discoverable at line 88 (ldconfig grep check) or at runtime.

Apply this diff to rebuild the cache:

  && echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
  && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf \
+ && ldconfig \
  && dnf clean all \

---

155-158: Critical: bitsandbytes 0.48.1 fails fail-fast check due to missing CUDA 12.8 wheels. Lines 157 pins bitsandbytes==0.48.1, which does not provide cu128-tagged wheel variants on PyPI. The fail-fast check will exit with status 1 during the build, preventing image completion. This issue was flagged in prior reviews but remains unresolved in the debug variant.

Upgrade to bitsandbytes==0.48.2 or later:

- RUN mkdir -p /tmp/wheels && \
-     python -m pip download --retries 5 --timeout 120 --only-binary=:all: --no-deps -d /tmp/wheels \
-       numba==0.62.1 bitsandbytes==0.48.1 || \
+ RUN mkdir -p /tmp/wheels && \
+     python -m pip download --retries 5 --timeout 120 --only-binary=:all: --no-deps -d /tmp/wheels \
+       numba==0.62.1 bitsandbytes==0.48.2 || \

Also update line 183 to match:

      tensorboard==2.19.0 \
-     bitsandbytes==0.48.1 \
+     bitsandbytes==0.48.2 \
      liger-kernel==0.6.2 \

---

75-78: Hadolint DL3044 violation: ENV statement self-references variables being defined. Lines 76–77 reference ${CUDA_HOME}, ${PATH}, and ${LD_LIBRARY_PATH} within the same ENV statement where they are being set, violating Docker best practice. Split into separate ENV statements to allow proper variable expansion.

Apply this diff:

-ARG CUDA_ARCH_LIST=9.0
-ENV CUDA_HOME=/usr/local/cuda \
-    PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH} \
-    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH \
-    TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ARG CUDA_ARCH_LIST=9.0
+ENV CUDA_HOME=/usr/local/cuda \
+    TORCH_CUDA_ARCH_LIST="${CUDA_ARCH_LIST}"
+ENV PATH=/usr/local/nvidia/bin:${CUDA_HOME}/bin:${PATH}
+ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${CUDA_HOME}/lib64:${CUDA_HOME}/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

images/universal/training/py312-cuda128-torch280/Dockerfile.debug (1)

232-316: Deterministic 2-step wheel build with comprehensive logging and timeout is a strong pattern. The build approach (lines 234–316) combines ccache caching, parallel compilation, detailed logging, heartbeat monitoring, and a 22-minute timeout with 120-second kill grace period. This significantly improves debuggability and prevents indefinite hangs. The inline bash script is verbose but appropriate for a debug variant.

As a future nice-to-have, consider extracting the long bash script into a separate shell file (e.g., build_mamba_wheel.sh) within the repository and COPY it into the image to improve readability and testability. This is not urgent for the current PR but would simplify future maintenance.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro